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Abstract 

The influence of a macroscopic time-dependent threshold on the retrieval dy- 
namics of attractor associative memory models with ternary neurons { — 1,0.+ 
1} is examined. If the threshold is chosen appropriately in function of the cross- 
talk noise and of the activity of the memorized patterns in the model, adapting 
itself in the course of the time evolution, it guarantees an autonomous function- 
ing of the model. Especially in the limit of sparse coding, it is found that this 
self-control mechanism considerably improves the quality of the fixed-point 
retrieval dynamics, in particular the storage capacity, the basins of attraction 
and the information content. The mutual information is shown to be the rel- 
evant parameter to study the retrieval quality of such sparsely coded models. 
Numerical results confirm these observations. 
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1. INTRODUCTION 



An important property required in efficient neural network modelling is an autonomous 
functioning independent from, e.g., external constraints or control mechanisms. For fixed- 
point retrieval by an attractor associative memory model this requirement is mainly ex- 
pressed by the robustness of its learning and retrieval capabilities against external noise, 
against malfunctioning of some of the connections and so on. Indeed, a model which em- 
bodies this robustness is able to perform as a content-adressable memory having large basins 
of attraction for the memorized patterns. Intuitively one can imagine that these basins of 
attraction become smaller when the storage capacity gets larger. This might occur, e.g., in 
sparsely coded models (Okada, 1996 and references cited therein). Therefore the necessity 
of a control of the activity of the neurons has been emphasized such that the latter stays the 
same as the activity of the memorized patterns during the recall process. In particular, for 
binary patterns with a strong bias some external constraints were proposed on the dynamics 
in order to realize this (Amit et al., 1987; Amari,1989; Buhman et al., 1989; Schwenker et 



An interesting question is then whether the recall process can be optimized without im- 
posing such external constraints, keeping the simplicity of the (given) architecture of the 
network model. To this end a self-control mechanism has been proposed in the dynamics of 
binary neuron models through the introduction of a time-dependent threshold in the trans- 
fer function. This threshold is determined in function of both the cross-talk noise and the 
activity of the memorized patterns in the network and adapts itself in the course of the time 
evolution (Dominguez and Bolle, 1998). 

The purpose of the present paper is to derive and verify this self-control mechanism 
for attractor networks with multi-state neurons. There are obvious reasons for choosing 
multi-state (or even analog) neurons in device oriented applications of neural networks. To 
give one example, the pixels of a colored or gray-toned pattern are represented by such 
neurons. In the sequel we restrict ourselves, for convenience and without loss of generality, 
to ternary neurons { — 1, 0, +1}. Although the dynamics and the interesting features of the 
latter have been discussed (see, e.g., Yedidia, 1989; Bouten and Engel, 1993; Bolle et al., 
1994 and references therein) especially in the limit of small pattern activity (Yedidia, 1989), 
no activity control mechanism has been proposed in the literature until now. 

The rest of this paper is organized as follows. In Section 2 we define the attractor associa- 
tive memory network model with ternary neurons and we introduce the relevant parameters 
in order to discuss the quality of the recall process. Section 3 introduces the mutual infor- 
mation function for this model and discusses an explicit analytic expression for it. In Section 
4 we study the fixed-point retrieval dynamics of the model with complete self-control. In 
particular, using a probabilistic signal-to-noise approach we obtain explicit time evolution 
equations and we consider the consequences of sparse coding. Section 5 studies the numeri- 
cal solutions of this self-controlled dynamics and compares the quality of the recall process 
in this case with the one for models without self-control. Numerical simulations are shortly 
discussed. Finally, in Section 6 we present some concluding remarks. 



Let us consider a network with N ternary neurons. At a discrete time step t the neurons 
Oi l <S {0, ±1}, i = 1, . . . ,N are updated synchronously according to the rule 



where h^t is usually called the "local field" (Hertz et al., 1991) of neuron i at time t. In 
general, the transfer function Fg t can be a monotonic function with 9 t a time-dependent 
threshold parameter. Later on it will be chosen as 



al., 1996). 



2. THE MODEL 
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p r~\ - J sign(x) if |x| > 6» f , > 

teAx) ~ \0 if |z| <0 4 . 1 j 

In the sequel, for theoretical simplicity in the methods used, the number of neurons N will 
be taken to be sufficiently large. 

The synaptic weights Jy are determined as a function of the memorized patterns £f £ 
{0, ±1}, i = 1, . . . , N, fj, = 1, . . . ,p, by the following learning algorithm 

In this learning rule the J(J are the standard Hebb weights (Hebb, 1949; Hertz et al., 
1991) with the ternary patterns taken to be independent identically distributed random 
variables (IIDRV) chosen according to the probability distribution 

p(«f)=m^-i)+(i-«)«(0- ( 4 ) 

Here a — (|£f | 2 ) is the activity of the memorized patterns which is taken to be the same for 
all \x and which is given by the limit N — oo of 



'N 



^Ei^r- (5) 



The brackets (• • •) denote the average over the memorized patterns. The latter are unbiased 
and uncorrelated, i.e., = 0, = 0. To obtain the Jjj themselves the Hebbian 

weights Jfj are multiplied with the Cy G {0, 1} which are chosen to be IIDRV with prob- 
ability Pr(dj = 1) = C/N,Pr{C tj = C jt ) = (C/N) 2 ,C/N «1,C > 0. This introduces 
the so-called extremely diluted asymmetric architecture with C measuring the average con- 
nectivity of the network (Derrida et al., 1987). 

At this point we remark that the couplings (|^) are of infinite range (each neuron inter- 
acts with infinitely many others) such that our model allows a so-called mean-field theory 
approximation. This essentially means that we focus on the dynamics of a single neuron 
while replacing all the other neurons by an average background local field. In other words, 
no fluctuations of the other neurons are taken into account, not even in response to chang- 
ing the state of the chosen neuron. In our case this approximation becomes exact because, 
crudely speaking, hi t is the sum of very many terms and a central limit theorem can be 
applied (Hertz et al., 1991). 

It is standard knowledge by now that synchronous mean-field theory dynamics can be 
solved exactly for these diluted architectures (e.g., Bolle et al., 1994). Hence, the big advan- 
tage is that this will allow us to determine the precise effects from self-control in an exact 
way. 

In order to measure the quality of the recall process one usually introduces the Hamming 
distance between the microscopic state of the network model and the fi — th memorized 
pattern, defined as 

d t = ]v E ~ a ' l ' t ' 2 = °^ ~ 2a N m m + 9m ■ (6) 

i 

This relation naturally leads to the definition of retrieval overlap between the fj, — th pattern 
and the network state 



l N,t 



Na N 



and the activity of the neurons, called neural activity 
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The nijy t are normalized parameters within the interval [—1,1] which attain the maximal 
value 1 whenever the model succeeds in a perfect recall, i.e., cr i: t — £f for all i. 

Alternatively, the precision of the recall process can be measured by the performance 
(Rieger, 1990; Shim et al., 1997) 

i 

which counts the relative number of correctly recalled bits. For subsequent manipulation, it 
is expedient to note that <% jCTi can be expressed as a linear combination of terms ^cr\ with 
k,l < 2 

^ = 1-^-^ + ^ + 1^. (10) 

Once the parameters (Q) and (@) are known, both these measures for retrieval can be cal- 
culated via the dynamics (0). Here we remark that for associative memory models with 
neurons having more than three states these measures for the retrieval quality can be de- 
fined in the same way. Then, technically speaking, the performance parameter (p|)-(^0|) will 
contain higher-order powers of <r£. 

Recently an information theoretic concept, the mutual information (Shannon, 1948; 
Blahut, 1990), has been introduced in the study of the quality of recall of some theoret- 
ical and practical network models (Dominguez and Bolle, 1998; Schultz and Treves, 1998; 
Nadal et al., 1998 and references therein). For sparsely coded networks in particular it turns 
out that this concept is very useful and, in fact, to be preferred (Dominguez and Bolle, 1998) 
above the Hamming distance. 

At this point we note that it turns out to be important to introduce the following quantity 
appearing in the performance 

1 N 

We call this quantity the activity-overlap since it determines the overlap between the active 
neurons and the active parts of a memorized pattern. Although it does not play any inde- 
pendent role in the time evolution of the associative memory model defined here it appears 
explicitly in the formula for the mutual information. 



3. MUTUAL INFORMATION 



In general, in information theory the mutual information function measures the average 
amount of information that can be received by the user by observing the signal at the 
output of a channel (Blahut, 1990). For the recall process of memorized patterns that we 
are discussing here, at each time step the process can be regarded as a channel with input 
and output o~ij such that this mutual information function can be defined as (forgetting 
about the pattern index /i and the time index t) 

I(o-i;6)=S(o-i)-(5((7 i |^)) e< ; (12) 

S(aMi)= -X>(cri|£0MpH6)]. (14) 

Here S(o~i) and S(o~i\£i) are the entropy and the conditional entropy of the output, re- 
spectively. These information entropies are peculiar to the probability distributions of the 
output. The term (S^Cil^j))^ is also called the equivocation term in the recall process. The 
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quantity p(<7j) denotes the probability distribution for the neurons at time t, while p{<Ji\£,i) 
indicates the conditional probability that the ith neuron is in a state <x; at time t, given that 
the ith pixel of the memorized pattern that is being retrieved is £j . Hereby we have assumed 
that the conditional probability of all the neurons factorizes, i.e., p({ci}|{£i}) = IL P^IC*); 
which is a consequence of the mean-field theory character of our model explained in Sec- 
tion 2. We remark that a similar factorization has also been used in Schwenker et al., 
1996. 

The calculation of the different terms in the expression (jl^) proceeds as follows. Formally 
writing (O) = (0) CT |C,£ = p(0 J2aP( a \0O f° r an arbitrary quantity O the conditional 
probability can be obtained in a rather straightforward way by using the complete knowledge 
about the system: (£) = 0, (a) = 0, (ag) = am, (£ 2 ) = a, (a 2 ) = q, (er 2 £) = 0, (<r£ 2 ) = 
0, (c 2 £ 2 ) = an, (1) = 1. The result reads (we forget about the index i) 

P(<r\Q= (*« + m&)K° 2 - 1) + (1 - 

- q-n 2 _q-an 

Sf= s - 4 , s = • (15) 

1 — a 1 — a 

Alternatively, one can simply verify that this probability satisfies the averages 

m =l((at)^)z, (16) 
q= «0«r|€>e> (17) 
n=~{{a 2 eUtk- (18) 



a 

These averages are precisely equal in the limit TV — * oo to the parameters m and q in (Q)-(]|) 
and to the activity-overlap introduced in ([ll]). Using the probability distribution of the 
memorized patterns (^), we furthermore obtain 

p(a) = 5>(£)?H£) - qS(a 2 - 1) + (1 - q)5(a). (19) 

The expressions for the entropies defined above become 

S(a) = -qlnl-(l-q)ln(l-q) (20) 

(S(a\0h = -(1 - <0 [*o In y + (1 - s ) ln(l - s )} 

n + m n + m n — m n — m 

— a \ — 7. — In — ^ 1 7. — In — 7. — 

1 2 2 2 2 

+ (1 - n) ln(l - n)] (21) 

These expressions are used in the next sections for discussing the quality of the recall 
process of our model with self-control dynamics. 



4. SELF-CONTROL DYNAMICS 



4.1 General equations 

It is standard knowledge (e.g., Derrida et al., 1987; Bolle et al., 1994) that the synchronous 
dynamics for diluted architectures can be solved exactly following the method based upon 
a signal-to-noise analysis of the local field (0) (e.g., Amari, 1977; Bolle et al., 1994; Okada, 
1996 and references therein). Without loss of generality we focus on the recall of one pattern, 
say fx = 1, meaning that only m l N t is macroscopic, i.e., of order 1 and the rest of the patterns 
cause a cross-talk noise at each time step of the dynamics. 

Supposing that the initial state of the network model {e>j,o} is a collection of IIDRV with 
mean zero and variance qo and correlated only with memorized pattern 1 with an overlap 
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mj it is wellknown (see the literature cited above) that the local field (|]) converges in the 
limit C, N — > oo to 

/ii,o = £>o + No] 1/2 JV(0,l) (22) 

where the convergence is in distribution and where the quantity 7V(0, 1) is a Gaussian random 
variable with mean zero and variance unity. The parameters m and q defined in the preceding 
sections have to be considered over the diluted structure and the (finite) loading a is defined 
by p — aC. 

This allows us to derive the first time-step in the evolution of the network. For diluted 
architectures this first step dynamics describes the full time evolution and we arrive at 
(Derrida et al., 1987; Yedidia, 1989; Bolle et al., 1993) 

/+oo 1 
Dz-ieFe^eml + iaq^z)) (23) 
-oo a 

/+oo 
DziFKeml + iaqt^z)). (24) 
-oo 

where we recall that the (• • •) denote the average over the distribution of the memorized 
patterns and Dz = dz[cxp(-z 2 /2)]/(2vr) 1 / 2 . 

Furthermore we also have the following expression for the activity-overlap 

nt+i = - / Dz{{e?Fl (em] + [aq t ] 1 / 2 z)) . (25) 

a J -co 

For the specific transfer function defined in (^) the evolution equations (23)-(pi[) reduce 

to 

,9t-m t d t +m t 
m t+1 =erjc{ — — =-) - er/c( — — =-) (26) 
y/aqt y/aqt 

r , f 0t-rnt , . f t 9 t +m t 
q t +i = a[erfc{ — — -) + erfc{ — =-) 
^/aqt y/Wt 

+ 2(l-a)[er/c(-^ = )], (27) 
where we have dropped the index 1 and with the function er/c(-) defined as 

1 r°° 

erfc(x) = -= J dz e~ z2/2 . (28) 



Without self-control these equations have been studied, e.g., in Yedidia, 1989 and Bolle et 
al., 1993. 

Furthermore the first term in ( p7| ) gives the activity-overlap. More explicitly 

9 t ~m t 9 t + m t 

n t+1 = erfc{ — ^^)+er/c( — . (29) 
y/aql ^/aqt 

Of course, it is known that the quality of the recall process is influenced by the cross- 
talk noise at each time step of the dynamics. A novel idea is then to let the network itself 
autonomously counter this cross-talk noise at each time step by introducing an adaptive, 
hence time-dependent, threshold of the form 

O f = c(a)y/aqi. (30) 

Together with Eqs.(5^)-(5q) this relation describes the self-control dynamics of the network 
model. For the present model with ternary neurons, this dynamical threshold is a macro- 
scopic parameter, thus no average must be taken over the microscopic random variables at 
each time step t. This is different from the idea used in some existing binary neuron models, 
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e.g., Horn and Usher, 1989 where a local threshold 9^ t is taken to study periodic behavior 
of the memorized patterns etc. Here we have in fact a mapping with a threshold changing 
each time step, but no statistical history intervenes in this process. 

This self-control mechanism is complete if we find a way to determine c(a). Intuitively, 
looking at the evolution equations (23)-(p4|) after the straightforward average over has 
been done and requiring that m ~ 1 — erfc(n s ) and q ~ a + erfc(n s ) with n s > inversely 
proportional to the error, the value of c(a) should be such that mostly the argument of the 
transfer function satisfies m — n sy /aq > 9 and n sy /aq < 9. This leads to c ~ n s . Here we 
remark that n s itself depends on the loading a in the sense that for increasing a it gets more 
difficult to have good recall such that n s decreases. But it can still be chosen a priori. 



4.2. Sparsely coded models 

In the limit of sparse coding (Willshaw et a!., 1969; Palm, 1980; Amari, 1989; Okada, 
1996 and references therein) meaning that the pattern activity a is very small and tends to 
zero for N — > oo it is possible to determine more precisely the factor c(a) in the threshold 

©• _ _ 

We start from the evolution equations (p6|)-(p7|) governing the dynamics. To have m ~ 1 

and n ~ 1 such that good recall properties, i.e., Oi — £j for most i are realized, we want 
m — ^faq 9. Activity control, i.e., q ~ a requires ^/aq <C 9. From Eq.(|30|) we then obtain 
1 <C c(a) <C -y= — 1. Then, for c(a) ^> 1 the second term on the right-hand side of Eq.(p^) 
leads to 

(1 - a)erfc[c{a)] - — L= exp[-^] (31) 



This term must vanish faster than a so that we obtain c(a) = y—2 ln(a). This turns out to 
be the same factor as in the model with binary neurons (Dominguez and Bolle, 1998). Very 
recently (Kitano and Aoyagi, 1998) such a time-dependent threshold has also been used in 
binary models but for the recall of sparsely coded sequential patterns in the framework of 
statistical neurodynamics (Amari and Maginu, 1988; Okada, 1995) with the assumption that 
the temporal correlations up to the intial time can be neglected in the limit of an infinite 
number of neurons. 

At this point we remark that in the limit of sparse coding, Eqs.(p6|)-(p9|) for the overlap 
and the activity-overlap become 

1 vn 

m t ~ n t ~ 1 - -erfc{—= - c)] (32) 

2 ^/aa 

Using all this and technically replacing the conditions -C above by <, meaning that we 
relax the requirement of perfect recall, we can evaluate the critical capacity for which some 
small errors in the recall process are allowed. We find 

a c = 0(\abx(a)\- 1 ), (33) 

which is of the same order as the critical capacity for binary sparsely coded network models 
with and without self-control (Tsodyks, 1988; Buhman et al., 1989; Perez- Vicente, 1989; 
Horner, 1989; Okada, 1996; Dominguez and Bolle, 1998). 

Next we turn to the quality of the recall process by the network. Because of the sparse 
coding the Hamming distance is not a good measure since it does not distinguish between a 
situation where most of the wrong neurons are inactive and a situation where these wrong 
neurons are active. The errors in recalling the active states are much more relevant since they 
contain more information. For instance, when Oj = for all neurons the Hamming distance 
d = a and vanishes in the limit of sparse coding, while when Oj = 1 for all neurons it is 
d = 1 — a and goes to 1. Clearly, in both cases no information is transmitted. Furthermore, 
suppose that all neurons are inactive, i.e., <7, = 0, then we have that m = 0, q — (and 



7 



n = 0), so the Hamming distance is d = a but there is no information transmitted. If 
instead of turning off the aN active neurons (meaning that = ±1) one would turn on aN 
neurons among the inactive ones (meaning that & = 0) one would get m — 1, q — 2a (and 
n = 1). So the Hamming distance is still d = a, but now some information is transmitted. 
It is intuitively clear that the first kind of action erases all the meaningful bits, while the 
second one does not affect essentially the code and, hence, leads to less important errors. 

In fact, we immediately note that for the first example I is, indeed, 0. In the second 
example we find that / = — aln(2a) — (1 — a) ln(l — a) = S(£) — 2aln(2) which is not much 
smaller than the entropy 5(£) of the memorized patterns. This confirms our statement that 
the mutual information is to be preferred above the Hamming distance for discussing the 
quality of recall by sparsely coded network models. 

5. NUMERICAL RESULTS 

Without selfcontrol the time evolution equations (|26|)-(|27|) have been studied in Yedidia, 
1989 and Bolle et al., 1994. Three different types of solutions were found. The zero solution 
determined evidently by to — and q — 0, a sustained activity solution defined by to = 
but q =/= and solutions with both to ^ and q =/= 0. There are both nonattracting 
and attracting solutions of the last type. It is straightforward to check that the mutual 
information is zero for both the zero solution and the sustained activity solution, since for 
the dynamics considered here, q = n whenever to = 0. Hence we restrict ourselves to the 
attracting solution, R with to > 0. 

We have solved this self-control dynamics (p6[)-(p7|) for our model in the limit of sparse 
coding and compared its recall properties with those of non-self-controlled models. 

The important features of the self-control are illustrated in Figs. 1-5. In Fig. 1 we have 
plotted the information content I a = pNI/#J = al as a function of the threshold 8 for 
a = 0.1 and a = 0.01 and different values of a, without self-control. This illustrates that 
it is rather difficult, especially in the limit of sparse coding, to choose a threshold interval 
such that I a is non-zero. We remark that these small windows for the threshold leading to 
non-zero information were used to determine what we call the optimal value of the threshold, 
6 opt, in Fig. 3. 

In Fig. 2 we compare the time evolution of the retrieval overlap, m t , starting from several 
initial values, Too, for the model with self-control and loading a = 3, an initial neural activity 
qo = 0.01 = a and 9 SC = [— 2(lna)ag t ] 1 / 2 , with the model where the threshold is chosen by 
hand in an optimal way in the sense that we took the one giving the greatest information 
content I a as seen in Fig. 1. We observe that the self-control forces more of the overlap 
trajectories to go to the retrieval attractor m = 1. It does improve substantially the basin 
of attraction. This is further illustrated in Fig. 3 where the basin of attraction for the whole 
retrieval phase R is shown for the model with a 6 op t selected for every loading a and the 
model with self-control 9 SC , with initial value go = 0.01 = a. We remark that even near the 
border of critical storage the results are still improved. Hence the storage capacity itself is 
also larger. These results are not strongly dependent upon the initial value of go a s long as 
g = 0(a). 

Figure 4 displays the information I a as a function of the loading a for the self-controlled 
model with several values of a. We observe that I max = Ia{o-max) is reached somewhat 
before the critical capacity and that it slowly increases with decreasing activity a. 

This is further detailed in Fig. 5 where we have plotted I m ax and a ma xo\ ln(a.) | as a 
function of the activity on a logarithmic scale. It shows that I m ax increases with | ln(a)| 
until it starts to saturate. The saturation is rather slow analogously to the model with 
binary neurons (Perez- Vicente 1989; Horner 1989; Dominguez and Bolle, 1998). 

Although we are well aware of the fact that simulations for such diluted models are difficult 
because the time evolution equations have been derived in the limit C, N — > oo with the well 
known condition C* << N 1 / 2 (Bolle et al., 1994), we have performed a limited number of 
numerical experiments. In this respect we note that it has recently been claimed (Arenzon 
and Lemke, 1994) in the study of binary neuron models that the analytic equation obtained 
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in the extreme dilution limit also fits very well those results (critical storage capacity and 
the overlap as a function of the loading a) obtained numerically for finite connectivity and 
under the less strong condition C « N (C ~ 20, TV - 16000) 

Typical results from our simulations are shown in Fig. 6. There we have plotted again the 
information content I a as a function of the loading a for a system with a = 0.1, C = 100, 200 
and N = 1 x 10 6 . Convergence to the retrieval attractor is obtained after maximum ten time 
steps. We compare the analytic results with the simulations for self-control and without 
self-control with a threshold taken to be 6 a — [— 2(lna)ago] 1 ^ 2 - Essentially, we clearly 
observe that self-control considerably improves the information content. For fixed loading 
the quantitative difference between theory and simulations is of order O — 1/ V Ca. Clearly, 
further numerical work is required but this falls outside the scope of the present study. 

6. CONCLUDING REMARKS 

In this paper we have introduced complete self-control in the dynamics of associative 
memory networks models with ternary neurons. We have studied the consequences of this 
self-control on the quality of the recall process by the network. To this purpose we have 
introduced the mutual information content for these models and shown that, especialy in 
the limit of sparse coding, this is a better measure for determining the quality of recall. 

We find that, exactly as in the binary neuron model (with static and with sequential 
patterns), the basins of attraction of the retrieval solutions are larger and the mutual infor- 
mation content is maximized. We have compared the analytic results with some simulations 
and we essentially confirm the improvement of the quality of recall by self-control. 

These results strongly suggest that this idea of self-control might be relevant for dynamical 
systems in general when trying to improve the basins of attraction and convergence times. 
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FIG. 1. The information I a as a function of 9 without self-control for a — 0.1 (top) and a — 0.01 
(bottom) for several values of a. 
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FIG. 2. The evolution of the overlap m t for several initial values mo, with qo = 0.01 = a and 
a — 3 for the model with self-control (right) and the optimal threshold model (left). The dashed 
curves are a guide to the eye. 
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FIG. 3. The basin of attraction as a function of a for a = 0.01 and initial qo = a for the 
self-controlled model (full line) and the optimal threshold model (dashed line) . 
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FIG. 4. The information I a as a function of a for the self-controlled model with several values 
of a. 
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FIG. 5. The maximal information I m ax and a max a\ ln(a)| as a function of | ln(a)|. 
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FIG. 6. The information I a as a function of the loading a for a — 0.1 and initial value qo = a 
with self-control and with fixed threshold 9q. Analytic results are compared with simulations for 
N = 10 6 and C = 100, 200. 
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